Added store_many_vectors on Mongo Storage #87
Conversation
There was a problem hiding this comment.
Could you add some tests for store_many_vectors to the mongo test suite?
| # Push JSON representation of dict to end of bucket list | ||
| self.mongo_object.insert_one(val_dict) | ||
|
|
||
| def _get_vector(self, hash_name, bucket_key, v, data): | ||
| """ |
There was a problem hiding this comment.
This docstring belongs to store_vector method
| def store_many_vectors(self, hash_name, bucket_keys, vs, data): | ||
| requests = [] | ||
|
|
||
| for v, d, bk in zip(vs, data, bucket_keys): |
There was a problem hiding this comment.
I suggest using from future.builtins import zip because it is more efficient in python2.7.
| def store_many_vectors(self, hash_name, bucket_keys, vs, data): | ||
| requests = [] | ||
|
|
||
| for v, d, bk in zip(vs, data, bucket_keys): |
| @@ -147,7 +166,7 @@ def get_bucket(self, hash_name, bucket_key): | |||
| shape=(val_dict['dim'], 1)) | |||
|
|
|||
| else: | |||
| vector = numpy.fromstring(val_dict['vector'], | |||
| vector = numpy.frombuffer(val_dict['vector'], | |||
There was a problem hiding this comment.
I updated this because I got some deprecation warnings
nearpy/storage/storage_mongo.py
Outdated
| {'lsh': {'$regex': self._format_hash_prefix(hash_name)}}) | ||
|
|
||
| def clean_all_buckets(self): | ||
| """ | ||
| Removes all buckets from all hashes and their content. | ||
| """ | ||
| self.mongo_object.remove( | ||
| self.mongo_object.delete_many( |
There was a problem hiding this comment.
remove method is deprecated so I replaced it with the suggested delete_many. It avoids annoying deprecation warnings
|
@etudor The test is broken because it cannot import |
|
@amorgun I have updated this |
|
@etudor It looks like a lot of mongo tests are broken now in python2.7. Please, check if it is related to your changes. Maybe you should pin an older version of |
Because I found this method to be very useful on Redis storage, I've added it to the mongo storage as well.
I haven't done a benchmark to compare what is the speed increase vs single inserts.